A proof of the data compression theorem of Slepian and Wolf for ergodic sources (Corresp.)

Let P(i) = (1 0)B’ be a peobability assignment on the set of nonnegative integers where 0 is an arbitrary real number, 0 < 0 < 1. We show that an optimal binary source code for this probabiliiy assignment is constructed as follows. Let I be the integer satisfying 0’ + @+ 1 < 1 < 8’ + @-I and represent each nonnegative integer i as i = b’+ r when j = [i/l], the integer part of i/l, and r = [i] mod 1. Encode j by a onary code (i.e., j zeros followed by a single one), and encode r by a Huffman code, using codewords of length /log2 11, for r < 2L’os 1+11 I, and length [log* I] -t1 otherwise. An optimal code for the nonnegative integers is the concatenation of those two codes. Theorem 2: M jointly ergodic countable-alphabet stochastic processes can be sent separately at rates R,,R,,. . .,R, to a common receiver with arbitrarily small probability of error, if and only if & R, 1 H(X”), i E S 1 X”), i E SC) = H(X'l' $2) 9 ,e. -,XcM’) H(X”), iESC) (16) for all subsets S cl {1,2,. . . ,M}, where SC denotes the complement of S, and H denotes the entiopy of the set of processes indexed by S conditioned on the processes indexed by SC. We have used the obvious extension of (2) to define the entropy H. Note, in the particular case M = 3, that these equations coincide with the following equations exhibited by Wolf [6] for the independent identically distributed case: The Huffman source coding algorithm [1], [2] is a wellknown algorithm for encoding the letters of a finite source alphabet into a uniquely decipherable code of minimum expected codeword length. Since the algorithm operates by successively “merging” the least probable letters in the alphabet, it cannot be directly applied ts infinite source alphabets. In this correspondence, we show how the Huffman algorithm can be used indirectly to prove the optimality of a code for an infinite alphabet if one can guess what the code should be first, Naturally it is not always easy to guess the structure of an optimal code, but if the structure is simple enough, and if one starts with the simplest cases, guessing oftefi works. The particular case that we deal with here is that of the nonnegative integers with a geometric probability assignment, R, 2 H(X”’ j X”‘, Xc3’) R, r H(XC2’ I X”‘, X(j)) R, 2 H(X’“’ j X”‘, XC2’) R1 + R2 2 H(X’I’, XC2’ 1 XC3’) R, + R3 z H(X”‘, XC3’ j X”‘) R, -tR3 2 H(X”‘, XC3’ 1 X”‘) P(i) = (1 8)8’, i>O (1) for some arbitrary 8, 0 < B c 1. This particular distribution arises in run-length doding, where if one has an independent letter binary source, with 8 being the probability of a zerb, then P(i) is the probability of a run of i zeros. The distribution also arises in other ways such as encoding protocol information in data networks. RI + R, + R3 2 H(X”’ X”’ XC3’) 3 5 * (17)